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Abstract 

This  paper  describes  the  activity  of  a  workshop  on  Data-Driven  Tradespace  Exploration  and  Analysis:  A  Key  Technical  Thrust  of 
Engineered  Resilient  Systems  (ERS).  The  workshop  was  attended  by  40  academic,  government,  and  industry  researchers  and 
practitioners  involved  in  tradespace  exploration  for  a  variety  of  engineering  domains.  The  one-and-one-half  day  workshop  sought 
to  develop  near  and  far  term  tradespace  technology  research  recommendations  for  the  ERS  Priority  Steering  Council  (PSC)  Lead. 
To  determine  promising  research  areas,  workshop  attendees  were  asked  to  describe  desired  tradespace  capabilities,  the  associated 
current  approach  and  its  deficiencies,  and  gaps  between  the  two  states.  These  research  areas  were  summarized  in  statements  of 
need,  supporting  rationale,  and  investment  timeframe.  Resilience  in  the  context  of  ERS  is  more  than  robustness;  resilience 
implies  that  when  the  system  is  placed  into  an  environment  in  which  it  was  not  originally  intended  to  operate,  after  some 
degradation  in  performance,  the  system  can  be  adapted  or  reconfigured  to  perform  at  its  intended  levels.  To  support  design  for 
resilience,  more  alternatives  must  be  generated  earlier,  considered  longer,  explored  over  multiple,  dynamic  alternative  futures, 
and  searched  exhaustively.  The  workshop  described  in  this  paper  was  organized  to  discuss  current  methods,  process,  and  tools  for 
performing  these  tradespace  analysis  related  tasks  and  to  better  understand  existing  tradespace  capabilities  and  their  suitability 
for  ERS. 
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1.  Background  and  Motivation 

In  April  of  2011,  the  Secretary  of  Defense  released  a  memorandum  describing  seven  priority  Science  and 
Technology  (S&T)  investment  areas  for  Fiscal  Years  (FY)  2013-2017^  One  of  these  investment  areas  was  identified 
as  ERS,  defined  as  “engineering  concepts,  science,  and  design  tools  to  protect  against  malicious  compromise  of 
weapon  systems  and  to  develop  agile  manufacturing  for  trusted  and  assured  defense  systems.”^  ERS  intends  to 
transform  the  engineering  of  complex  systems  to  make  the  resulting  systems  more  affordable,  effective,  and 
adaptable.  The  intent  is  to  enable  engineers  and  decision  makers  to  focus  on  the  ‘right’  things  during  system 
development  such  that  the  end-product  system  is:  1)  effective  in  a  wide  range  of  operations  and  2)  effective  across 
multiple  alternative  futures. 

At  the  time  this  workshop  was  conducted,  the  ERS  priority  area  consisted  of  five  key  technical  thrusts,  one  of 
which  was  Data-Driven  Tradespace  Exploration  and  Analysis.  Upon  transfer  of  the  ERS  PSC  Lead  role  from  the 
Office  of  the  Secretary  of  Defense  (OSD)  to  the  U.S.  Army  Engineer  Research  and  Development  Center  (ERDC)  in 
late  2012,  the  focus  of  the  tradespace  thrust  shifted  slightly,  from  supporting  four  other  key  technical  thrusts,  to 
being  the  foundation  for  ERS  tradespace  analytics  and  decision  support^.  Tradespace  analysis  for  ERS  initially  was, 
and  remains,  concerned  with  the  generation  of  large  numbers  of  alternative  designs,  early  in  the  design  process,  and 
the  evaluation  of  these  multi-attribute  designs  across  multiple  dimensions  such  that  more  informed  decisions  can  be 
made.  Resilience  in  the  context  of  ERS  is  more  than  robustness;  resilience  implies  that  when  the  system  is  placed 
into  an  environment  in  which  it  was  not  originally  intended  to  operate,  after  some  degradation  in  performance,  the 
system  can  be  adapted  or  reconfigured  to  perform  at  its  intended  levels.  To  support  design  for  resilience,  more 
alternatives  must  be  generated  earlier,  considered  longer,  explored  over  multiple,  dynamic  alternative  futures,  and 
searched  exhaustively.  In  generating  and  evaluating  these  alternatives  throughout  the  design  process,  the  concept  of 
resilience  is  critical. 

The  workshop  described  in  this  paper  was  organized  to  discuss  current  methods,  process,  and  tools  for 
performing  these  tradespace  analysis  related  tasks  and  to  better  understand  existing  tradespace  capabilities  and  their 
suitability  for  ERS.  A  summary  of  workshop  outcomes  and  tradespace  challenges  has  been  previously  presented  to 
the  systems  engineering  community^.  The  current  document  provides  workshop  outcomes  and  recommendations. 

2.  Tradespaces  for  Engineered  Systems 

A  tradespace  is  a  multidimensional  solution  space  in  which  information  about  design  alternatives  is  displayed  in 
order  to  allow  decision  makers  to  investigate  how  much  capability  in  one  metric  must  be  given  up  in  exchange  for  a 
specified  capability  in  another  metric  (i.e.,  tradeoff),  and  to  understand  how  much  capability  must  be  forfeited  in 
order  to  achieve  a  feasible  solution  (i.e.,  compromise).  A  tradespace  is  a  repository  of  qualitative  and  quantitative 
information  that  can  be  queried,  filtered,  and  visualized  in  order  to  help  reveal  information  to  a  person  making  a 
decision"^.  By  visualizing  the  information  in  different  ways  (e.g.,  through  rotation,  translation,  coloring,  and  sizing  of 
graphical  representations),  decision  makers  can  identify  trends  and  features  that  reveal  the  most  advantageous 
design  solutions  in  terms  of  the  trades  that  may  need  to  be  made. 

The  ERS  tradespace  is  not  envisioned  simply  to  be  populated  with  more  design  alternatives  and  additional 
metrics  as  compared  to  a  traditional  tradespace;  it  is  envisioned  to  be  interactive,  to  be  capable  of  being 
supplemented  with  new  information  in  real-time  as  the  design  process  proceeds,  and  to  include  criteria  not 
traditionally  used  in  early-phase  design-decision  making  because  of  inherent  uncertainty  or  insufficient  knowledge. 
The  ERS  tradespace  is  envisioned  to  be  collaborative,  persistent,  updated,  and  consulted  throughout  the  system 
lifecycle.  It  is  expected  to  be  a  common  interface,  between  multiple  decision  makers  at  multiple  levels  and  stages  of 
the  design  and  development  process,  that  indicates  possible  alternative  systems,  the  compromises  required  for 
achieving  them,  and  the  effects  of  decisions  made. 

Historically,  many  individuals  and  organizations  have  indicated  that  they  perform  “tradespace  exploration”  for 
different  purposes  and  using  different  techniques.  The  term  “tradespace”  therefore  has  different  meanings  to 
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different  people.  From  the  technologists  and  engineers,  to  the  program-level  decision  maker,  and  to  the  user,  the 
amount  of  technical  detail  needed  to  inform  a  decision  varies.  The  amount  of  information  to  be  included  in  the 
tradespace  therefore  needs  to  be  tailored  based  on  the  conditional  and  subjective  perspectives  of  multiple  decision 
makers^.  A  significant  goal  of  the  ERS  tradespace  workshop  was  to  discuss  tailoring  tradespaces  to  the  needs  of 
different  decision  stakeholders. 

3.  Workshop  Objectives 

The  primary  workshop  objective  was  to  capture  the  state  of  the  art  and  the  gaps  in  tradespace  exploration  (TSE) 
and  analysis  capability  (e.g.,  theory,  methods,  processes,  tools).  Participants  were  asked  to  consolidate  the  identified 
gaps  into  actionable  technology  research  areas,  which  would  be  used  to  guide  tradespace  research  in  support  of 
enabling  ERS  as  by  the  PSC  Lead.  A  secondary  objective  was  to  identify  opportunities  for  collaborative  research  in 
the  area  of  tradespace  studies.  Questions  for  discussion  included; 

•  How  are  tradespace  studies  conducted  with  current  methods,  processes,  and  tools? 

•  What  are  the  needs  for  improved  tradespace  studies  that  would  enable  ERS  goals? 

•  How  essential  are  each  of  these  needs  and  how  far  are  we  from  achieving  each  of  them? 

•  Where,  and  when,  should  we  be  investing  to  achieve  each  of  these  needs? 

4.  Workshop  Format 

The  invitation-only  workshop  was  targeted  for  academia,  government,  and  industry  involvement,  including 
experts  in  optimization,  modeling  and  simulation  (M&S),  data  visualization,  value-focused  thinking,  decision 
making,  and  trade  studies.  Eour  Armed  Services  were  represented  amongst  the  40  participants.  The  workshop  was 
conducted  at  the  Potomac  Institute  for  Policy  Studies  in  Arlington,  VA,  and  the  U.S.  Army  Research  Laboratory 
(ARE)  planned  and  moderated  the  workshop  agenda. 

The  workshop  was  conducted  over  one-and-one-half  days.  Day  1  included  introductory  briefings  from  the  ERS 
Priority  Area  Lead  and  the  ARE  Tradespace  Technical  Thrust  Lead,  as  well  as  introductory  briefings  from  breakout 
session  chairs  in  order  to  provide  a  basis  for  discussion  during  subsequent  breakout  sessions.  Day  1  concluded  with 
preliminary  reports  from  the  breakout  session  chairs.  Day  2  was  dedicated  to  conducting  breakout  sessions,  and 
concluded  with  briefings  from  each  group  and  a  workshop  wrap-up  discussion. 

The  workshop  was  held  in  conjunction  with  a  Tradespace  and  Affordability  Program  (TAP)  workshop  sponsored 
by  the  Systems  Engineering  Research  Center  (SERC)^.  The  purpose  of  the  TAP  workshop  was  to  bring  together 
researchers  from  various  SERC  collaborator  universities.  Department  of  Defense  (DoD)  sponsors,  potential 
transition  partners,  and  other  experts  in  the  tradespace  and  affordability  area,  with  the  primary  objective  of 
establishing  the  near-term  direction  for  the  SERC’s  TAP  initiative.  The  definition  of  the  TAP  tradespace  was 
specific  to  exploring  how  to  better  create  and  analyze  the  tradespace  of  “ilities”  (e.g.,  affordability,  flexibility, 
reliability,  usability,  and  interoperability).  It  was  envisioned  that  the  TAP  and  ERS  tradespace  efforts  would  inform 
each  other  and  establish  a  basis  for  continued  cooperation. 

5.  Workshop  Proceedings 

5.1.  Introductions 

Dr.  Robert  Hummel,  Chief  Scientist,  Potomac  Institute  for  Policy  Studies  provided  a  welcome  and  overview  to 
all  attendees.  He  described  the  mission  of  the  non-profit  Potomac  Institute  for  Policy  Studies  as  public  policy 
research  that  integrates  business  and  government  interests  into  meaningful  options. 

Dr.  Robert  Neches,  Director,  Advanced  Engineering  Initiatives,  Office  of  the  Deputy  Assistant  Secretary  of 
Defense,  Systems  Engineering/ERS  PSC  Lead,  provided  an  overview  of  the  ERS  priority  area^.  Although  all  of  the 
attendees  had  received  public  release  briefing  materials  on  ERS  and  Tradespace  Analysis  with  their  invitations,  not 
all  attendees  were  intimately  familiar  with  ERS.  This  briefing  provided  a  succinct  overview  of  the  background. 
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motivation,  end  state  description,  and  a  possible  development  path  for  ERS.  Dr.  Neches  explained  that  the  ERS  PSC 
has  collected  information  on  nearly  50  related  research  programs  across  DoD,  however  there  are  gaps  that  these 
programs  are  not  addressing.  Dr.  Neches  went  on  to  state  that  TSE  technology  solutions  need  to  be  identified  for  the 
purpose  of  closing  the  gaps,  in  order  to  achieve  the  vision  of  ERS.  He  provided  a  list  of  major  tenets  of  ERS:  earlier, 
better-informed  decision  making;  better  data  repositories,  deeper  analyses,  and  more  options  considered  longer;  a 
balance  between  performance,  cost,  and  capability;  using  hardware  and  sensors  to  learn  user  needs;  and 
implementing  use  cases  that  are  generated  from  user  interests  and  needs. ^ 

Mr.  Elias  Rigas,  Chief,  Vehicle  Applied  Research  Division,  ARL/ERS  Data-Driven  Tradespace  Exploration  and 
Analysis  Thrust  Area  Lead,  provided  an  introduction  to  the  ERS  Tradespace  thrust.  The  briefing  discussed  TSE 
within  the  context  of  ERS,  as  well  as  the  workshop  goals.  Mr.  Rigas  stressed  that  the  tradespace  thrust  of  ERS  must 
aid  in  communicating  to  decision  makers  what  they  are  investing  in,  and  where  they  should  be  investing. 
Additionally,  the  ERS  tradespace  needs  to  be  expanded  to  include  the  tacit  information  that  decision  makers  use, 
such  as  affordability,  manufacturability,  and  reliability. 

5.2.  Breakout  Groups 

Attendees  were  assigned  to  individual  break  groups  that  were  formed  based  on  perceived  functions  of  TSE.  Prior 
to  the  workshop,  the  lead  for  each  of  the  breakout  groups  was  invited  to  prepare  a  10-minute  introduction  of  their 
breakout  area.  The  intent  was  to  give  the  breakout  leads  the  ability  to  mold  the  breakout  sessions  around  how  they 
envisioned  the  problem,  as  well  as  provide  an  informal  overview  of  each  breakout  area  to  all  attendees.  In  this  way, 
all  attendees  would  be  exposed  to  the  background  and  main  discussion  points  encountered  by  all  groups.  This  also 
gave  the  attendees  an  opportunity  to  submit  questions  and  topics  for  breakout  sessions  in  which  they  were  not 
scheduled  to  participate. 

5.2.7.  Broadening,  Populating,  and  Managing  the  Tradespace 

Dr.  Art  Pyster,  Deputy  Director  of  the  SERC,  provided  his  perspective  on  broadening  the  tradespace.  Dr.  Pyster 
posited  that  the  premise  of  being  able  to  select  the  “best”  choice  in  a  tradespace  assumes  that  the  tradespace  contains 
a  sufficient  number  of  feasible/viable  design  alternatives  as  measured  by  the  criteria  against  which  the  alternatives 
will  be  ranked,  scored,  or  selected.  An  important  question  is  whether  sufficient  information  exists  to  adequately 
describe  alternatives  in  the  tradespace,  and  whether  this  information  is  stable,  complete,  robust,  integrated,  and  even 
appropriate  for  quantitative  or  qualitative  analysis.  With  multiple  stakeholders,  each  with  differing  attribute 
preferences,  there  is  a  need  to  ensure  that  necessary  and  sufficient  information  is  included  in  the  tradespace.  Dr. 
Pyster  asked  his  group  to  think  about  qualitative  data  in  a  tradespace  and  what  it  ‘looks  like’  (e.g.,  security,  policy, 
safety).  Although  decision  makers  use  this  information  in  their  decision  making  process,  there  are  challenges 
pertaining  to  how  these  broad  qualitative  concepts  can  be  included  in  the  tradespace  in  a  manner  that  would  allow 
either  humans  or  machines  to  evaluate. 

5.2.2.  Linking  the  Tradespace 

Dr.  Christina  Bloebaum,  Dennis  and  Rebecca  Muilenburg  Professor  of  Aerospace  Engineering  at  Iowa  State 
University,  provided  her  perspective  on  linking  the  tradespace  to  the  analysis/design  environment  and  to  the 
decision  making  environment.  Although  there  is  a  desire  to  have  the  tradespace  communicate  bi-directionally  with 
data-generating  models  and  the  decision  making  environment,  there  are  technical  and  policy  challenges  to  doing  so. 
Some  of  these  challenges  include  incomplete,  inconsistent,  and  incompatible  data  provided  from  multiple  sources  at 
multiple  times.  Dr.  Bloebaum  asked  her  group  to  think  about  mechanisms  for  establishing  links  into  and  out  of  the 
tradespace,  and  how  these  links  could  enable  more  effective  TSE. 

5.2.3.  Searching,  Exploring,  and  Analyzing  the  Tradespace 

Dr.  Brian  German,  Assistant  Professor  in  Aerospace  Engineering  and  Associate  Director  of  the  Aerospace 
Systems  Design  Laboratory  (ASDL)  at  the  Georgia  Institute  of  Technology,  provided  his  perspective  on  searching 
the  tradespace.  Dr.  German  posed  a  question  of  how  do  humans,  viewing  a  graphical  representation  of  tradespace 
data,  know  when  a  feature  or  tradeoff  worthy  of  further  analysis  has  been  discovered.  There  are  fundamental 
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questions  of  what  outcomes  are  being  sought  -  an  optimum  (maximum/minimum),  a  single  solution,  a  family  of 
solutions,  points  of  discussion,  areas  of  compromise,  or  other  features.  Dr.  German  asked  his  group  to  think  about 
how  solutions  such  as  search  methods,  novel  graphics,  and  classification  algorithms  could  be  applied  to  TSE. 

5.2.4.  Acting  on  the  Tradespace 

Dr.  Adam  Ross,  Lead  Research  Scientist  in  the  Systems  Engineering  Advancement  Research  Initiative  of  the 
Massachusetts  Institute  of  Technology,  provided  his  perspective  on  acting  on  the  tradespace.  He  pointed  out  that 
decision  makers  may  come  to  consensus  on  the  best  design  option,  but  that  this  decision  is  based  on  the  current 
scenario.  Dr.  Ross  asked  his  group  to  consider  how  changing  operational  contexts  can  affect  the  makeup  of  the 
tradespace,  and  ultimately  the  decision;  changing  operational  contexts,  lifecycle  changes,  and  technology 
improvements  can  change  a  design  currently  described  as  ‘bad’  into  a  ‘good’  design.  If  it  were  possible  to  fully 
explore  these  possibilities,  planning  could  then  be  tailored  around  them.  Dr.  Ross  also  presented  his  group  with  a 
challenge  stemming  from  two  types  of  decision  makers  involved  in  acting  on  the  tradespace:  technical  leadership, 
who  desire  a  holistic  view  of  the  system,  and  financial  leadership,  who  desire  highest  cost-benefit  and  probability  of 
success  concepts. 

5.2.5.  Modeling  and  Simulation ’s  Role  in  Generating  Tradespace  Data 

Ms.  Philomena  Zimmerman,  Deputy  Director  of  Modeling,  Simulation  and  Analysis  in  the  Office  of  the 
Assistant  Secretary  of  Defense,  provided  her  perspective  on  an  area  pertinent  to  all  four  breakout  areas  (though  was 
not  a  breakout  area  on  its  own):  a  common  system  description  and  design  model.  If  such  a  common  system  model 
existed  there  could  be  consistency  and  re-use  of  data  objects,  which  would  create  consistency  across  TSE.  Rather 
than  new,  flat  models  being  created  to  represent  new  instances  of  components  and  then  linked  across  subsystems 
and  domains,  formal  and  rich  models  would  be  drawn  from  a  repository  (i.e.,  reused)  to  build  a  cohesive 
representation  of  what  the  system  is  and  does.  With  this  approach  the  system  model  would  be  dynamic  and  usable  in 
simulations  as  opposed  to  just  being  a  mechanism  for  generating  static  data  points. 

6.  Breakout  Group  Summaries 

The  primary  output  of  this  workshop  is  the  recommendations  put  forth  by  each  breakout  group,  phrased  as 
concise  research  areas  of  interest.  Although  the  overall  topic  area  name  is  necessarily  an  aggregate  description,  the 
supporting  rationale  and  the  observations  together  provide  sufficient  detail  to  define  the  research  areas.  It  is 
expected  that  these  recommendations  will  serve  as  reference  points  for  the  current  ERS  leadership  to  initiate 
tradespace  research  activities  in  the  near  and  far  term.  The  technology  investments  were  binned  into  near-  (1-3 
years),  mid-  (3-5  years),  and  far-  (5-7  years)  term  needs. 

6.1.1.  Broadening,  Populating,  and  Managing  the  Tradespace 

The  Broadening,  Populating,  and  Managing  Breakout  Group  identified  a  total  of  six  key  technology  investments, 
all  of  which  are  near-term  needs.  Each  is  discussed  briefly  here. 

6. 1.1.1.  Characterizing  the  “ilities”  (BPMl) 

Data  characterizing  the  "ilities"  of  tradespace  options  is  often  missing  or  inadequate,  even  though  the  quality  of 
those  ilities  has  a  material  impact  on  the  relative  value  of  the  options  to  the  stakeholders.  Research  is  needed  in:  (a) 
methods  and  tools  for  integrating  the  analysis  of  physical,  informational,  and  human  characteristics  of  systems;  (b) 
standardization  of  “ility”  terminology  and  concepts  to  facilitate  communication  among  stakeholders;  and  (c) 
languages  and  tools  to  express  and  analyze  ilities  in  quantitative  terms  that  can  be  understood  by  diverse 
stakeholders  so  they  can  state  their  preferences  and  understand  the  implications  of  those  preferences.  With  such 
languages  and  tools,  it  would  also  be  easier  to  establish  incentives  and  monitor  how  well  contractors  are  realizing 
ilities  during  system  development.  Developers  need  to  hire  staff  with  requisite  skills,  grow  organically,  or  outsource. 
Researchers  need  to  create  more  intuitive  tools  and  methods,  develop  agreed  upon  terminology  to  express  ilities,  and 
build  cross-impact  models  and  relationships  that  quantitatively  determine  the  impact  of  changing  one  ility  on  the 
other  ilities. 
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6. 1.1.2.  Future  Environments  and  Scenarios  (BPM2) 

Anticipated  and  possible  future  operational  environments  and  user  scenarios  are  usually  not  stated  in  sufficient 
detail  to  support  tradespace  evaluation.  Research  is  needed  to  create  general  toolkits  to  rapidly  assemble  concepts  of 
operations  in  multiple  domains  in  a  way  that  vividly  communicates  to  diverse  stakeholders  and  can  be  translated 
into  parameters  for  a  model  of  the  system.  Milestone  decision  authorities  should  insist  on  exploring  possible  future 
scenarios.  Systems  Engineering  practice  should  be  involved  in  very  early  phases  of  the  lifecycle,  in  parallel  with 
S&T  activities.  For  example,  applying  holistic  system  cost  models  to  justify  greater  expenditures  early  in  lifecycle 
for  the  purpose  of  reducing  total  cost  of  ownership  (TOC).  Research  should  be  conducted  to  develop  better  ways  to 
understand  which  constraints  drive  the  value  of  the  system,  measuring  robustness  at  the  scenario  level,  and 
diversifying  those  who  evaluate  scenarios. 

6. 1.1. 3.  Affordability  (BPM3) 

Affordability,  an  ility  worthy  of  special  note,  is  often  not  addressed  during  trade  studies  (which  often  focus  on 
technical  aspects  of  trades  apart  from  the  cost  implications  across  the  full  lifecycle,  including  manufacturing, 
operations,  and  maintenance).  Better  TOC  methods  and  tools  are  needed  that  account  for  system  of  systems,  avoid 
suboptimization  on  colors  of  money  across  the  lifecycle,  and  avoid  suboptimization  around  different  phases  of  the 
lifecycle  such  as  Sc&T,  development  and  manufacturing,  and  support.  As  these  methods  and  tools  emerge,  they 
should  be  applied  in  contracts. 

6. 1.1.4.  Missing  Data  (BPM4) 

The  government  often  doesn't  require  contractor  data  to  be  submitted  in  formats  that  facilitate  analysis  or 
comparisons  across  alternative  designs  (e.g.,  using  standard  formats  or  engineering  models).  The  government  should 
require  submission  in  standard  formats  that  are  amenable  to  automated  analysis  and  include  more  intellectual 
property  (IP)  clauses  into  contracts. 

6. 1.1. 5.  Tradeoffs  Not  Considered  (BPM5) 

The  tradeoff  of  greater  risk  vs.  greater  performance,  lifecycle  cost,  or  another  desirable  system  characteristic  is 
often  not  properly  considered.  The  government  and  developers  should:  (a)  provide  technology  maturation  funds  to 
advance  lower  readiness  level  technology  up  to  levels  6  and  7;  (b)  recognize  that  technology  readiness  levels  (TRL) 
are  system  dependent;  and  (c)  compare  alternative  technologies  using  effectiveness  models  at  the  mission  level  such 
that  systems  are  simulated  in  their  intended  operational  environments. 

6. 1.1.6.  Historical  Knowledge  (BPM6) 

Historical  knowledge  that  could  inform  the  creation  and/or  evaluation  of  options  is  often  not  captured  and  made 
readily  accessible  at  the  enterprise  level,  making  it  difficult  to  recognize  and  apply  that  knowledge  when  needed. 
Limited  access  to  previously  developed  knowledge  and  other  artifacts  such  as  engineering  models  and  system 
architectures  result  in  regeneration  of  data  rather  than  reuse  of  existing  data. 

6.1.2.  Linking  the  Tradespace 

The  Linking  Breakout  Group  identified  a  total  of  six  key  technology  investments,  with  four  near-term  (L1-L4) 
and  two  mid-term  needs  (L5-L6). 

6. 1.2.1.  Standard  Interchange  Methodology  (LI ) 

The  participants  recognized  that  the  need  to  link  the  tradespace  capability  to  design  and  decision-making 
environments  should  be  bi-directional.  Not  only  should  these  environments  inform  the  tradespace,  but  the  tradespace 
should  inform  the  environments.  In  order  for  this  to  become  practical,  however,  it  is  recognized  that  a  standard 
interchange  methodology  must  exist  to  provide  for  the  greatest  inclusion  of  externally  provided  or  available  tools,  as 
well  as  data.  Such  a  methodology  should  be  scalable  to  address  any  size  of  tradespace,  as  well  as  the  needs  of  the 
different  stakeholders  and  decision  makers.  It  is  critical  that  such  an  interchange  methodology  be  platform 
independent,  enabling  the  greatest  breadth  of  tools  to  be  supported. 
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6. 1.2.2.  Knowledge  Management  Infrastructure  (L2) 

Separate  from  the  Interchange  Methodology  listed  in  LI  is  the  need  for  an  infrastructure  that  will  provide  for 
rigorous  treatment  of  data  storage,  interchange  and  reuse  for  knowledge  being  generated  as  well  as  used.  The  needed 
knowledge  management  infrastructure  should  facilitate  the  ability  to  link  across  tools  and  environments.  Such  an 
infrastructure  would  provide  for  longevity  of  data  and  knowledge,  which  would,  given  trust  and  confidence  factors 
as  well  as  uncertainty  representations,  enable  use  of  the  data  over  the  long  term  despite  any  changes  in  personnel  or 
organizational  reorganization.  Additionally,  such  a  knowledge  management  infrastructure  will  enable  the  linking  of 
the  elements  supporting  a  decision  to  the  actual  decision  being  made. 

6. 1.2. 3.  An  Engineering  Modeling  Framework  (L3) 

At  present,  there  is  no  formal  mechanism  for  ensuring  information  integrity  from  the  earliest  stages  of  the  design 
process  through  to  the  later  stages,  and  then  back  again.  In  order  to  achieve  the  desired  bi-directional  linking  of 
design  and  decision  environments  to  the  tradespace  environment,  a  framework  must  exist  to  provide  for  a  unifying 
approach  that  is  used  across  all  participating  groups  in  the  process. 

6. 1.2.4.  Conceptual  Design  Toolbox  (14) 

At  present,  the  conceptual  design  stage  of  a  large-scale  engineered  system  development  process  relies  heavily  on 
a  tradespace  capability.  It  is  at  this  stage  that  the  greatest  flexibility  exists  to  make  meaningful  trades  in  a  system’s 
context.  Yet,  there  are  no  standardized  methods  or  tools  to  support  the  trades  necessary.  The  common  tools  used 
today  are  often  discipline-specific  and  rarely  enable  an  understanding  of  how  decisions  will  impact  the  larger  system 
preference.  Having  a  mechanism  to  enable  quick  assessment  of  ‘goodness’,  with  an  ability  to  scale  to  large  systems, 
would  be  extremely  advantageous.  Having  the  ability  to  link  these  modeling  capabilities  to  those  that  would  be  used 
in  detailed  design  stages  is  also  critical.  This  would  better  enable  a  seamless  propagation  of  modeling  and  simulation 
from  early  stages  in  the  process  through  to  the  later  stages,  and  vice  versa. 

6. 1.2. 5.  System  Architecture  Representation  (L5) 

It  is  recognized  that  organizational  structure  often  drives  the  system  architecture,  resulting  in  an  undue  influence 
on  the  outcomes  of  a  system  product.  Hence,  a  need  exists  for  a  system  architecture  that  will  maintain  the 
representation  of  the  system  regardless  of  organizational  structure.  Such  an  architecture  should  then  enable  bi¬ 
directional  trades  for  all  decision  makers  in  the  process  (across  all  levels  of  stakeholders)  as  well  as  across  the 
lifecycle  of  the  system. 

6. 1.2.6.  Tradespace  Consistency  (L6) 

Different  stakeholders  will  require  different  information  at  different  times  in  the  lifecycle  of  a  system.  Hence,  the 
tradespace  environment  should  provide  a  seamless  consistency  and  continuity  across  all  stakeholders,  with  the 
appropriate  level  of  fidelity  being  automatically  identified  and  supplied.  This  consistency  is  particularly  important  to 
enable  the  bi-directional  linking  desired  for  the  tradespace. 

6.1.3.  Searching,  Exploring,  and  Analyzing  the  Tradespace 

The  Searching,  Exploring,  and  Analyzing  Breakout  Group  identified  a  total  of  10  key  technology  investments, 
with  three  near-term  (SEA3,  SEA7,  SEA9),  two  near-to-mid-term  (SEAl,  SEAS),  one  mid-term  (SEAS),  one  mid- 
to-far-term  (SEAIO),  two  far-term  (SEA4,  SEA6),  and  one  need  spanning  the  near-far  spectrum  (SEA2). 

6. 1.3.1.  Communicating  Tradespace  Data  (SEAl ) 

Guided  TSE  with  decision  makers  in  the  loop.  Capabilities  are  needed  to  illustrate  design  decisions  and  the 
impact  of  constraints/objectives  in  interactive  sessions.  This  interaction  capability  should  be  able  to  queue  analyses 
with  linked  tools  on  demand  and  in  near  real-time  based  on  questions  by  decision  makers.  Desire  interactive 
scenarios  and  storylines  that  anticipate  the  interests  of  decision  makers,  show  traceability  in  previous  tradespace 
analysis,  and  indicate  how  recommended  designs  were  found  by  design  engineers  and  technologists.  Approaches  to 
design  and  automatically  produce  effective  summary  graphics/reports  for  decision  makers. 
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6. 1.3.2.  Interactive  Visualization  (SEA2) 

Effective  visual  representations  of  time  series  data  for  dynamics  in  the  physical  system  and  for  changes  in  the 
tradespace  over  time.  Graphics  that  efficiently  and  effectively  present  n-D  tradespace  data  by  going  beyond  the 
state-of-the-art  in  current  2-D  and  3-D  views.  Innovative  interaction  paradigms  that  can  identify  and  manipulate 
sampled  designs  within  the  tradespace  more  holistically  than  traditional  control  paradigms  such  as  slide  bars  and 
brushing.  Intuitive  visualizations  must  be  readily  internalized  by  the  viewer,  efficiently  communicating  meaning. 

6. 1.3. 3.  Preserving  Tradespace  Search  Decisions  and  Rationale  (SEA3) 

Customizable  and  automated  features  to  record  the  tradespace  search  activity  (e.g.,  paths  taken  by  the  user 
through  the  tradespace,  time  stamps  and  user  annotations,  audio/video  recording).  Ideally,  these  features  should 
allow  users  to  record  tradespace  sessions  in  a  way  similar  to  screen  video  capture  software,  but  with  much  richer 
tradespace-specific  features.  Post-processing  algorithms  and  interfaces  would  allow  a  user  to  refine  these  recordings 
into  a  presentation  format  for  documenting  the  design  process.  Additionally,  the  capability  could  be  used  to  develop 
recordings  to  educate  engineers  on  best  practices  in  TSE. 

6. 1.3.4.  Incorporating  the  “ilities”  (SEA4) 

Methods  to  facilitate  TSE  and  valid  decision  making  when  metrics  such  as  “ilities”  are  involved.  The  ability  to 
understand  the  impact  of,  or  characterize  the  uncertainty  in,  using  such  “ilities”  in  the  design  process  could  aid  in 
streamlining,  potentially  reducing  the  risk  of  over -designing  only  to  have  an  “ility”  invalidate  previous  decisions. 

6. 1.3.5.  Search  and  Sampling  Algorithms  (SEAS) 

Effective  and  efficient  search  and  sampling  methods  that  better  span  the  space  and/or  zoom  to  particular  regions 
at  a  specified  or  pragmatic  computational  cost.  Multi-fidelity  search  approaches  to  leverage  more  rapid  analysis 
tools  while  tracking  and  managing  error  with  higher  fidelity  tools.  Dynamic  emulation  and  identification  of  primary 
processes  to  inform  search  and  bridge  the  gap  between  fidelity  tiers.  Mechanisms  are  needed  to  guide  searches  based 
on  selective  dimensions,  while  still  tracking  other  dimensions. 

6. 1.3.6.  Identification  of  Tradespace  Features  (SEA6) 

Approaches  to  filter  and/or  identify  interesting  areas  of  large  tradespaces  prior  to  or  during  interactive 
exploration.  Novel  classification  and  pattern  recognition  algorithms  that  allow  humans  to  identify  abstract  and 
previously  unknown  objectives  and  constraints  by  interaction  with  tradespace  graphics.  Similar  algorithms  could 
help  to  guide  search  to  interesting  regions  based  on  observations  of  past  search  behavior  by  the  designer  and/or  a 
more  informal  specification  of  the  search  objectives  than  with  classical  optimization  approaches. 

6. 1.3.7.  Applying  Uncertainty  to  the  Tradespace  (SEA7) 

Eormal  understanding  of  the  relevant  types  of  uncertainty  that  are  involved  in  the  tradespace  problem  and 
systematic  approaches  to  model  and  mitigate  these  uncertainties.  Identification,  attribution,  and  characterization  of 
sources  of  uncertainty  are  crucial  to  this  process.  Uncertainties  also  imply  additional  tradeoffs  (e.g.,  one  may  pay 
more  to  achieve  less  variability  in  system  performance  through  additional  expenditures  during  the  design  process  or 
by  designing  additional  robustness  into  the  system).  There  is  a  need  for  research  into  paradigms  for  representing  and 
managing  tradeoffs  based  on  uncertainty. 

6. 1.3.8.  Applying  Formal  Decision  Making  Models  to  TSE  (SEAS) 

Descriptive,  prescriptive,  and  normative  decision  making  models  for  TSE  activities.  Eormal  processes  that  seek 
to  drive  irrationality  and  misalignment  of  incentives  out  of  the  decision  making  process.  Characterization  of  how 
TSE  leads  to  decision  making  in  the  engineering  of  complex  systems. 

6. 1.3. 9.  Characterizing  and  Trading  Resilience,  Robustness,  Flexibility,  and  Adaptability  (SEA9) 

Eormal  domain-specific  and  domain-independent  metrics  to  define  and  assess  resilience,  robustness,  flexibility, 
and  adaptability  of  engineered  systems  for  use  in  TSE  and  decision  making.  Models  to  evaluate  and  trade  these 
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metrics  relative  to  the  overall  system  requirements  and  value  proposition;  if  a  goal  of  ERS  is  the  design  of  resilient 
systems,  then  resiliency  must  be  quantified,  modeled,  and  included  in  the  system  tradespace. 

6.1.3.10.  Systematic  Appending  and  Reduction  of  the  Tradespace  (SEAIO) 

Multi-step  approaches  to  manage  the  propagation  of  design  decisions  and  the  tradespace  itself  over  time  during 
system  development.  Specific  desired  capabilities  include;  (a)  archiving  tradespaces  at  each  phase  in  the  systems 
engineering  process;  (b)  systematically  narrowing  the  set  of  designs  under  consideration  using  new  knowledge;  (c) 
identifying  optimal  recourse  actions  and  points  of  reversion  to  prior  decision  points;  (d)  systematically  quantifying 
and  reducing  uncertainty  at  each  phase;  (e)  illustrating  the  reduction  of  the  available  tradespace  as  decisions  are 
incrementally  locked  in  and/or  as  requirements  change;  and  (f)  valuation  and  options  pricing  formulations  as  an  aid 
to  decision  making  at  each  phase  in  the  development  process. 

6.1.4.  Acting  on  the  Tradespace 

The  Acting  Breakout  Group  identified  a  total  of  14  key  technology  investments,  grouped  here  into  four  areas, 
with  two  near-term  (ACTl,  ACT3),  one  near-to-mid-term  (ACT4),  and  one  spanning  the  near-far  spectrum  (ACT2). 
The  individual  investment  area  components  developed  at  the  workshop  are  indicated  with  an  “A#”. 

6. 1.4.1.  TSE  Foundations  (ACTl;  Al,  A2,  All,  A16) 

TSE  Eoundations  seeks  to  establish  foundations  for  purposefully  conducting  TSE  to  support  decision  making. 
Al;  If  tradespaces  are  expected  to  be  the  basis  for  design  decisions,  then  it  is  critical  to  understand  what  these  key 
decisions  are.  A  set  of  key  decisions  should  be  developed  that  allow  for  focused,  purposeful  execution  and 
continuous  improvement  of  TSE.  A2;  Previously  conducted  TSE  can  provide  insights  and  lessons  as  leverage 
towards  understanding  how  TSE  actions  and  decisions  have  impacted  historical  systems.  Understanding  these  links 
can  reveal  the  impact  that  formal,  consistent  TSE  can  have  on  a  system.  A12;  Build  upon,  rather  than  reinvent, 
knowledge  gained  from  early  phase  TSE.  Apply  the  knowledge  gained  from  prior  decisions  by  using 
formal/informal  codification  of  lessons  and  insights,  and  use  this  as  leverage  throughout  the  lifecycle  and  across 
different  programs.  A16;  As  tradespaces  are  to  be  the  foundation  for  decisions,  as  well  as  the  repository  for  decision 
rationale,  then  fundamentals  must  be  established  for  conducting  TSE.  Eurther,  there  must  be  advocacy  for  the 
development  of  TSE  education  and  training;  rather  than  continued  dependence  on  portions  of  TSE  research  and 
knowledge  scattered  across  multiple  domains,  fuse  this  content  into  its  own  core  capability. 

6.1. 4.2.  TSE  Process  (ACTl;  A4,  A5,  A6,  A9,  A15) 

TSE  Process  seeks  to  establish  best  practice  for  TSE.  A4;  Interrogate  and  update  tradespaces  within  the  context 
of  a  seminar  or  workshop  to  develop  group  decision  maker  buy-in.  Interactive  sessions  with  real-time  feedback  are 
expected  to  increase  trust  of  the  tradespace  and  therefore  increase  confidence  in  TSE-based  decisions.  A5;  Desire 
the  ability  for  TSE  to  be  useful  even  with  incomplete  information.  Nontechnical  factors  and  unarticulated  objectives 
are  often  omitted  from  the  tradespace,  but  are  influential  in  decision  making.  A6;  Guidelines  are  needed  for  knowing 
when  one  has  performed  “best  practice”  TSE.  Varied  formalism  in  TSE  tools,  processes,  and  execution  due  to 
lifecycle  phase  and  system  complexity  must  be  overcome.  A9;  Desire  a  mechanism  for  conferring 
legitimacy/accountability  to  TSE  to  allow  for  incorporation  in  existing  acquisition  process.  A15;  Desire  a 
formalized,  iterative,  sequential  TSE  process  to  avoid  decisions  based  on  trial-and-error  TSE. 

6. 1.4. 3.  TSE  for  Resilience  (ACT3;  Al,  All) 

TSE  for  Resilience  seeks  to  establish  ways  to  explicitly  consider  system  resilience  in  the  tradespace.  A  resilient 
system  solution  is  no  longer  just  materiel  -  it  includes  strategies  for  sustainment.  A7;  Evaluate  system  alternatives 
across  appropriate  types  and  quantities  of  alternative  futures,  and  discover  which  scenarios  “break”  the  system.  All; 
Means  for  generating,  evaluating,  and  valuating  a  set  of  adaptability  approaches  with  designs  to  deliver  value 
robustness  and  resiliency  across  alternative  futures. 
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6. 1.4.4.  TSE  Throughout  the  Lifecycle  (ACT4;  A3,  A8,  AlO,  A13,  AM) 

TSE  Throughout  the  Lifecycle  seeks  to  establish  the  tradespace  as  a  decision  enabler.  A3:  Consistency  and 
compatibility  of  TSE  activities  throughout  the  lifecycle,  eliminating  inconsistent  assumptions  and  making  previous 
tradeoff  insights  both  transparent  and  capable  of  being  revisited.  A8:  Trace  preferences  and  decisions  across  the 
lifecycle  and  the  system  hierarchy.  Understand  how  sequential  TSE-based  decision  making  limits  the  system  value 
achievable.  AlO:  Counter  increasingly  large  tradespaces  by  using  human  expertise  and  metrics  as  appropriate 
“filters”  for  effective  decision  making.  A13:  Use  TSE-based  decisions  to  help  set  requirements  and  also  understand 
how  requirements  drive  system  value.  A14:  Desire  for  TSE  to  support  multi-stakeholder  collaboration,  creating  the 
ability  to  quickly  identify  and  focus  on  conflicts  and  confluence  among  decision  makers. 

7.  Workshop  Conclusions  and  Next  Steps 

Many  methods,  processes,  and  tools  have  been  developed  to  generate,  store,  manipulate,  search,  and  act  upon 
data  in  a  decision  making  environment.  However,  the  process  of  performing  tradespace  analysis  for  exploring  the 
realm  of  feasibility,  and  demonstrating  that  sufficient  knowledge  of  the  system  has  been  generated  for  decision 
making,  is  inconsistent  and  has  little  reuse  across  multiple  tradespace  explorers.  Issues  of  uncertainty  propagation, 
subjective  data,  high-dimensional  spaces,  static  and  flat  visualizations,  combinatorial  what-if  scenarios,  feature 
identification,  and  retention  of  data  throughout  a  lifecycle  have  not  been  sufficiently  addressed.  These  and  other 
gaps  partially  exist  because  there  appears  to  be  interchangeability  in  the  lexicon  of  related  terms  such  as  trade-off 
study,  optimization,  alternative  analysis,  and  value-focused  thinking.  The  breakout  group  results  presented  here 
show  that  there  is  necessarily  some  amount  of  overlap  in  the  defined  breakout  group  functions,  evident  by  repetition 
in  needs.  This  bolsters  assumptions  that  TSE  is  a  collaborative  effort  in  support  of  the  decision  making  process 
throughout  a  system’s  lifecycle,  there  is  no  independent  function  of  “perform  tradespace  exploration”,  and  the 
responsibility  to  support  decision  making  is  shared  across  those  who  generate,  store,  search,  and  act  on  data. 

The  research  agenda  presented  here  is  intended  to  illuminate  gaps  and  challenges,  for  the  purpose  of  advancing 
research  in  tradespace  analysis  methods,  processes,  and  tools  from  their  current  capability  to  that  desired  for 
realizing  the  ERS  vision.  It  is  expected  that  government,  industry,  and  academic  research  organizations  will  refer  to 
these  gaps  and  challenges  to  purposefully  align  their  tradespace  exploration  research  efforts. 
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